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ABSTRACT 

A survey of public library users was conducted 
regarding retrieval results of unassisted user searches in general 
periodical indexes. The data were analyzed to evaluate the impact or 
lack of impact of searching a CD-ROM index with loosely-controlled 
vocabulary and expanded-search capabilities (i.e., Inf oTrac-General 
Periodicals Index) versus a print index with a more highly-controlled 
vocabulary (i.e., Readers' Guide to Periodical Literature). 
Statistics include the age of the searcher, the experience with 
searching indexes, the experience with computers, recall and 
precision retrieval results, and user satisfaction with retrieval 
results. Problem areas considered in InfoTrac were variant headings 
assigned for one individual and one heading assigned to two different 
people. The results of the study indicate higher user satisfaction 
with the electronic index even when recall and precision rates are 
lower than when using the print index. Overall results indicate 
users* precision scores are lower when multiple variant headings are 
used with no cross-references and that users are not aware of the 
inaccuracies of their searches. The user questionnaire is included in 
the appendix. (Contains 24 references.) (Author/ JLB) 
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Retrieval Results of Unassisted Searches of Two General 

Periodical Indexes 

A survey wrs conducted regarding retrieval results of 
unassisted-user searches in general periodical indexes* The data 
was analyzed to evaluate the impact or lack of impact of searching 
a CD-ROM index with a loosely-controlled vocabulary versus a print 
index with a highly-controlled vocabulary. Statistics include the 
age of the searcher , the experience with searching indexes , the 
experience with computers, recall and precision retrieval results, 
and user satisfaction with retrieval results* 

Problem areas considered in InfoTrac were variant headings 
assigned for information about the one individual and one heading 
assigned to two different people* The results of the study 
indicate higher user satisfaction with the electronic index, 
InfoTrac, even when recall and precision rates are lower than when 
using the print index, Readers ' Guide to Periodical Literature * 
This could be a result of the perception that the database was 
probably larger and that the user missed relevant articles. 
Overall results seem to indicate users' precision scores are lower 
when multiple variant headings are used with no cross-references, 
and that users' are not aware of the inaccuracies of their 
searches . 
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1 . Introduction 



Authority control has been a basic principle of librarianship 
for decades. Whether in the card catalog or as a part of an index, 
authority control used in conjunction with a system of cross- 
references has provided the means for gathering together 
information on like subjects and providing access to that 
information through various access points. In the practice of 
indexing the term authority control may not be commonly used but 
the term consistency is. Consistency itself is at the core of 
authority control. 

As we move into the future with capabilities for boolean or 
keyword searching in electronic databases, the value of maintaining 
established subject headings and strict authority control has been 
discussed. Along with this trend is the rapid move from print 
indexes to electronic indexes. Computer-aided indexing provides 
an opportunity for improvement in consistency of access to 
information for these users. The resulting product of computer- 
aided indexing has been seen to fall anywhere along a continuum; 
some are better than print indexes and some are worse. 

Reference librarians, though squeezed between the budget cuts 
resulting from an economic recession and the difficulty of keeping 
up with the increasing use of libraries, still voice concern over 
the lack of quality in indexes and databases. It is, in most 
cases, considered the responsibility of the information staff to 
determine policies and to determine which tools are appropriate for 
individual searches (Booth and South 1982). There has been an 
increase in interest in streamlining services in public as well as 
academic libraries so that more unassisted research can take place. 
In a public library the researcher may be a student, businessman, 
consumer, or job seeker. The indexes used are usually general 
periodical indexes . 



1.1 The Problem 

The problem addressed by this study was to examine the impact 
or lack of impact of maintaining traditional authority control over 
subject headings in a general periodical index. To reach this end, 
the study examined the impact on an unassisted, untrained user's 
search when searching a CD-ROM index with loosely-controlled 
vocabulary and expanded-search capabilities versus a print index 
with a more traditionally-structured vocabulary (access points). 
Specifically, the study examined the results of the user's search 
for records in Inf oTrac-General Periodicals Index containing 
perceived problem headings versus the results of the same search 
in the print index Readers' Guide to Periodical Li terature. 
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1.2 Definitions of Terms 



Authority control in this study is defined as "the process of 
maintaining consistency in the verbal form used to represent an 
access point and the further process of showing the relationships 
among names, works, and subjects" (Taylor 1992). 
IAC is an acronym for Information Access Company of California. 
InfoTrac is a trademark of the Information Access Company of 
California. The label InfoTrac refers to several CD-ROM indexing 
and abstracting databases offered by IAC. Some of these are 
LegalTrac, Health Reference Center, and General Periodicals Index. 
The term InfoTrac in this study refers only to the General 
Periodicals Index, available since 1985. 

Precision is the proportion of retrieved items that are relevant 
and recall is the proportion of relevant items retrieved as a 
result of a search (Salton 1992). According to Losee (1991), when 
R and N represent the number of relevant and non-relevant documents 
in the database and R r . t and N r . t represent the number of relevant and 
non-relevant documents already retrieved, these measures may be 
computed as follows: 

R rwt and _ . . R,»«t 

Recall = r~ Precision = (Rrwt + Nrat ) . 

Rowley (1982) states these formulas as: 

Number of relevant documents retrieved x 10 q% 

Recall ratio - numbeir of relevant documents in the system 

. . . _ Number of relevant documents retrieved x 10 o% 

Precision ratio - Total number of documents retrieved 



Readers' Guide to Periodical Litera ture is a general periodicals 
print index produced by H, W. Wilson Company and available since 
1900. 

Unassisted research refers to patrons receiving no assistance or 
training from library staff to use the bibliographic databases on 
CD-ROM (Allen 1989). Hall, Talan, and Pease ( 1987) mention "naive 
students' using InfoTrac with little instruction or guidance; Reese 
(1988) mentions that InfoTrac is a "self-explanatory index for 
beginners" and that many users have no previous library research 
experience. In this study, the phrase unassisted research or user 
will refer to the user receiving no instructions in the use of the 
index other than those provided by the index itself. In addition, 
the user will be provided with no instruction from the library 
staff or the researcher in developing search techniques. 
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1.3 Limitations of the Study 

Since the monthly InfoTrac CD-ROM spans the years 1986 to the 
previous month of the current year, the years searched in this 
study included 4.986 to the month currently included in the InfoTrac 
database. The data collection was conducted over no longer that 
two consecutive weeks for InfoTrac as InfoTrac is updated monthly. 
Because the number of records included in a database that spans 
eight years is considerable, three sample searches with a limited 
number of possible retrievals were chosen to test the success of 
the users' searches. The study was limited to two months for data 
collection, compilation and evaluation of the results. The numbers 
of individuals willing to participate was small and the rejection 
rate was high; therefore, the total sample size was 44. 

2. Literature Review 
2.1 Overview of Indexing Quality 

In the field of indexing and abstracting there are many 
elements to consider when looking at quality. The elements of 
quality in an index may not be easily enumerated as defining them 
would involve subjective processes and because indexes are designed 
for different purposes and for different users. 

Jennifer Rowley believes that "An index, whatever theory it 
reflects, must be comprehensible to its users". She gives a 
synopsis of features that a good index would exhibit, the major 
points being that the subject and the relationships between 
subjects are labeled effectively so that search strategies can be 
developed by the user. Other researchers have mentioned elements 
of quality in an online database record that could also be 
measured in a CD-ROM database. These are mechanical accuracy, 
appropriateness of headings, level of detail and intellectual 
accuracy (Zeng 1993). 

As a facet of information retrieval and storage, the quality 
of indexes and indexing has been measured by various tests. Some 
consider Armand V. Feigenbaum's Total Quality Control to be the 
standard in determining quality in indexes (Buchan 1992). Though 
the tests for quality in indexes have not been officially 
standardized the most widely used and known tests are recall and 
precision (Losee 1992; Salton 1992), which are frequently connected 
to user satisfaction (Chu and Ajiferuke 1989). 

Chu and Ajiferuke (1989) indicate that relevance and user- 
satisfaction judgments create problems because of subjectivity. 
The relevance formulas are based on the concept of exhaustivity in 
which it is assumed that the searcher wants to find all records 
related to the focus of the search. Some researchers believe that 
this is rarely the case (Salton 1992), though it is really up to 
the searcher to decide which items are relevant. If not all 
documents are indexed with appropriate cross-linkj ng so that the 
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records can be found , then the searcher's option to determine 
relevance is lost before he even begins. 

Other methods of measurement of quality of online systems have* 
been developed for the National Library of Medicine (White and 
Griffith 1987) and proposed for use by White and Griffith (1987) • 
In this method , three dimensions of an indexing system are 
identified for measurement. These are "(a) linking related 
documents , (b) discriminating broadly among these linked subsets 
within the entire file, and (c) discriminating finely among 
individual documents." 

Whichever method is used, there are recurring variables being 
studied with each of the methodologies — exhaustivity through 
linking of documents and discrimination through specificity. With 
the increasing use of computers in the Information Age, the 
possibility of providing true consistency in indexes is very real. 
Early on some information specialists felt that automatic indexing, 
in which the indexing was done entirely by the computer, would be 
the wave of the future. In reality it seems that the most current 
viable use of computers in indexing is computer-aided indexing, in 
which the computer is used to manipulate text and to maintain 
consistency and standards through lexicographical interaction 
(Buchan 1992) . 

A good index should provide precision, consistency and cross- 
linking in order to provide access to information. In this way the 
phrase "A mis-shelved book is a lost book" applies to poor 
indexing. Without quality indexing, the information is lost. 

The opportunity for lost information is especially noticeable 
especially when related to the indexing of forms of names. Texts 
on indexing or information filing agree that it is extremely 
important to link related documents (White and Griffith 1987) when 
associated with variant forms of personal names (Wellisch 1991; 
Booth and South 1982). One of the most useful methods of assuring 
comprehension of an index to a user is to link documents indexed 
under variant names of the same person with the traditional 'see 
also' reference. In addition to this, a descriptor added to a name 
entry will facilitate continued consistency in the name headings 
by eliminating ambiguity of identity and the resulting confusion 
on the part of the searcher and the indexer. 



2.2 InfoTrac and 
Readers' Guide to Periodical Literature 

Since authority control and accessibility are intertwined, the 
opportunity for error in identifying relevant entries is considered 
to be especially high for the untrained information seeker (i.e. 
one without much practice). The untrained information seeker is 
most likely to use a general periodical index since it is multi- 
disciplinary, and this user is likely to come from all walks of 
life, seeking various kinds of information. The two most commonly 
found general periodical indexes found in academic and public 
libraries are InfoTrac and Readers' Guide to Periodical Literature . 
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InfoTrac - General Periodical* Tndex is a periodical index on 
CD-ROM produced by Information Access Company (TAC) of California, 
IAC also produces related products such as Health Reference Center. 
From this point, all references to InfoTrac will refer only to the 
General Periodicals Index. Readers 9 Guide to Periodical Literature 
is a well-known print index published by H.W. Wilson. Both indexes 
are commonly found in public and academic libraries , are used by 
students and the general public , and use Library of Congress 
subject headings. 

Since the introduction of InfoTrac in 1985 , most literature 
about it has related to user acceptance and satisfaction- Almost 
over-whelmingly positive , feedback from users has been visual as 
well as verbal. Though extremely popular (Van Arsdale and Ostrye 
1986; Beltran 1987; Momenee 1987; Hall, Talan and Pease 1987; Jaffe 
1988; Reese 1988; Ernest and Lange 1989; and Forrest, Chapman and 
Wright 1989) , InfoTrac has raised concerns from librarians who 
question the value of the searches performed by unassisted users. 
In some cases, the concerns about quality are over the apparent 
lack of awareness of the inappropriateness of the database for 
their particular subject search. In other cases, though users seem 
satisfied, librarians have not been convinced that the quality of 
the authority control in the indexing has allowed the user to 
retrieve appropriate citations. Librarians have a professional 
responsibility to provide quality resources and popularity is not 
necessarily a bottom-line criteria for providing a particular 
resource (Van Arsdale and Ostrye 1989). 

In a 1986 article in Database, Richard D. Carney, a Vice 
President of Information Access Company, described InfoTrac as a 
"self-contained periodical reference system designed for in-library 
use by library patrons". In the same article, he focuses on the 
evaluation of reference tools and the effect the tool will have on 
information specialists who face increasing demands on their time. 
All over the country library staff are coping with fall-out from 
the economic recession: budgets are cut but use of libraries is up. 
Therefore the tool that Mr. Carney describes is particularly 
appealing, especially when he emphasizes, " The key is to employ 
the new technology to facilitate more unassisted research by 
patrons". Since InfoTrac is promoted as a database for beginners 
and users appear to be abandoning print indexes, succumbing to the 
lure of the def initiveness of a computer system, the inaccuracies 
and inadequacies in the authority files of this widely-used index 
should be of concern • 

Past studies indicate that patrons/users c|o participate in 
unassisted research and feel satisfied with their search results 
(Momenee 1987; Jaffe 1988; Forrest, Chapman and Wright 1989). The 
majority of the studies were done in academic settings where it was 
found that InfoTrac was a tool most appropriately and 
enthusiastically used by undergraduates (Beltran 1987). Jaffe at 
Sweet Briar College in Virginia found that students preferred the 
controlled-index search method of InfoTrac rather than the boolean 
free-text searching of WILSONDISC. Both Jaffe and Beltran found 
that the multi-disciplinary approach of InfoTrac was an advantage 
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when used by undergraduates because the students tended to believe 
that the results of their searches were definitive even ir the 
database they had used was not appropriate for the subject 
searched. This finding makes Carney's offering that the system is 
self-contained and facilitates more unassisted research by patrons 
worrisome, since the purpose of the information specialist is to 
assist in locating the most accurate, reliable and comprehensive 
information available within the constraints of the available 
resources. Reese conducted a study comparing Readers' Guide to 
Periodical Literature with InfoTrac but the controls fell short of 
those needed to validate the results (Reese 1988). 

InfoTrac is indexed with Library of Congress subject headings. 
Occasionally, additional topical subject headings are added when 
the new headings have not yet appeared in the Library of Congress 
system. According to the IAC handbook, those new headings created 
by IAC are linked to the nearest LC subject heading by a cross- 
reference. Headings in the alphabetically-arranged subject index 
may be topics; personal, product or company names; and titles of 
books, movies, plays, etc. All variant forms of an individual's 
name will appear in the subject listing in the alphabetic position 
where it would naturally fall. In the case of names this 
researcher searched, there were no cross references to variant 
forms of the name. The heading takes the form of the name as it 
appeared in the entry. 

This approach is directly opposite the findings in the 
literature concerning features of a good index in that there is a 
lack of consistency in the vocabulary, a lack of 'linking related 
documents', and a lack of discriminating finely among individual 
documents. In theory this should produce a high failure rate when 
untrained searchers use the system. Some entries may be found with 
'boolean' searching. In 1986, Carney felt that boolean 

searching was not effective for "a reference product designed to 
serve large numbers of untrained users". Since then, the 
capability for an 'expanded search' has been added creating system- 
driven 'and' searches. Some information specialists feel that 
boolean capabilities replace the need for a controlled vocabulary 
with authority control. This study is to examine the results of 
searches performed by untrained InfoTrac users when the searches 
are based upon records with perceived problem headings. 



3 . Methodology 

A survey was conducted involving public library users in a 
suburban metropolitan area. Groups of users performed pre-selected 
search queries in two general periodical indexes. A random sample 
of forty-four public library users were asked to participate 
through accidental sampling of those who asked the reference staff 
for assistance with an information query. It was assumed that the 
majority of those who asked for assistance in the public library 
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would be untrained users since trained users (i.e. those who do 
research frequently in information databases and understand the 
structure of those databases) typically do not request assistance. 
It was assumed that the general periodical indexes being studied 
are designed for use by a wide range of people with a diversity of 
interests, therefore anyone approaching the reference staff with 
a reference query may have justification for use of those indexes. 

,?5 e f a ^ a collecte <* were the numbers of relevant citations 
identified by untrained, unassisted users from a pre-selected set 
perceived to contain problem access points or headings. The 
questionnaire and data collection form used was a variation of the 
form used by Reese in 1987 at the Brookdale Community College 
Learning Resources Center. At that time, her sample included a 
final total of 17 students which was insufficient to draw anv 
serious conclusions. 1 



3.1 Procedures and Design 

Two groups of results are compared in the study concerning the 
usability and quality of two separate general periodical indexes 
when used by untrained, unassisted searchers. Voluntary 
participation was asked of patrons who approached the reference 
staff with information questions. The study was conducted by the 
researcher who asked the first patron to do a search in InfoTrac- 
General Periodicals index 1986-current. The next patron was asked 
to do the identical search in the volumes of Wilson's Readers' 
guide to Periodical Literature that covered the same time period. 
The only assistance given in the search was an explanation of what 
subject headings look like in each of the indexes and how to 
identify the citations following headings that meet the 
specifications of the search. 

Each searcher was given a data collection form which included 
a user profile section. No names were recorded. The user profile 
included age, past index use, past computer use, and frequency of 
use of any indexes, databases, or computers. 

Each searcher was asked to search one question so that each 
question was to be searched by eight to 23 people on InfoTrac and 
eight to 23 people in Readers' Guide. The final range of 
participants was five to nine people for each search. The purpose 
of the study was to see how well the untrained searcher handles the 
problem of variant forms of names, the problem of an absence of 
descriptors for individuals sharing the same headings and an 
absence of cross-references in these cases. Attached to each data 
collection form was a photocopy of a title page of a book related 
to the search question. The searcher was asked to record each 
subject heading or search method used, even if no heading or 
entries were found, and to record the citations found that fit the 
requirements of the search. Recording the citations on InfoTrac 
may be done by pressing the letter 'p' or by pressing the 'print' 



key. Recording the citations in Readers ' Guide required hand 
writing the subject heading used, the author of the article, the 
journal name and date, and the volume and page number of the index. 
If no entries were found, that was to be noted. Also, the searcher 
was asked if he received any outside assistance and from whom. 

This study was designed to examine the retrieval success of 
users of two general periodical indexes when the sample population 
is unassisted by trained individuals. Comparison of the results 
when using InfoTrac, an electronic database, and when using 
READERS' GUIDE TO PERIODICAL LITERATURE, a traditionally-designed 
print index, is made to indicate the impact or lack of impact of 
searching an index constructed with a loosely-controlled vocabulary 
and which uses no descriptors. 

In a preliminary search by the researcher, it was found that 
there were 6 variations of the name W. EDWARDS DEMING in the 
InfoTrac (1986-August 1993) database. All of these entries applied 
to the same individual , founder of TQM management principles and 
author of OUT OF THE CRISIS. No cross-references or qualifiers 
were given for any of the variations. The variations were: 

Deming , Edwards 

Deming, Edwards W. 

Deming , W . Edward 

Deming , W . Edwards 

Demming, Edward 

Demming , Edwards . 

For the name of Robert Waller, there were at least four variant 
headings. The entries applied to at least two and possible three 
different individuals. No qualifiers, descriptors or cross 
references were given. One entry, WALLER, ROBERT, combined 
articles about two different individuals: 1. the author of THE 
BRIDGES OF MADISON COUNTY , and 2. a British writer on industrial 
and political relations. The variations in the name entries were: 

Waller, Robert 

Waller, Robert A. 

Waller, Robert J. 

Waller, Robert James. 

When searching for information on Kate Smith, singer and author of 
UPON MY LIPS A SONG, the researcher found one heading for the name 
SMITH, KATE. There were 10 entries under this one heading. The 
first two applied to an author/writer of current information. The 
latest article by this Kate Smith appeared in January 1990. The 
other eight articles applied to the singer Kate Smith, who died in 
1986. There, again, were no descriptors or cross-references as all 
Kate Smiths were listed under the same heading. 

When looking through a comparable range of years in READERS' 
GUIDE TO PERIODICAL LITERATURE 1986-current , the researcher found 
one variation of the name SMITH, KATE. All articles listed under 
this heading were about the singer who died in 1986. There was 
one variation of Deming 's name from 1986 to current: 
DEMING, W. EDWARDS. 
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There were two variations of the author Robert Waller's name as 
well as a heading for a second Robert Waller. The 1990 volumo 
listed: 

WALLER, ROBERT, 1913- . 
The 1991 and 1992 volumes listed: 

WALLER, ROBERT J. 
In 1993 there was a qualification added to WALLER, ROBERT J. to 
expand it to: 

WALLER, ROBERT JAMES, 1939- . 
The articles and headings found by the researcher in READERS' GUIDE 
are included in Appendix C. 



4. IMPLEMENTATION 

Six variations of questionnaire packets were created and 
distributed by the researcher in the reference area of a medium- 
sized public library. The searches were for information about 
three authors: Kate Smith, Robert Waller, and Edwards Deming. Each 
search was paired to have been searched in InfoTrac and in READERS' 
GUIDE TO PERIODICAL LITERATURE in order to compare recall and 
precision results. Volunteers were solicited from individuals who 
approached the reference area. Library users were told that the 
study was to determine the effectiveness and 'user-friendliness' 
of two magazine indexes in the library. 

The survey began the last of August 1993. The researcher 
proceeded with the survey during various times of the day and week 
in order to survey a range of people in different lifestyles. The 
proposed period of time for the completion of the survey was a two- 
week period. At the end of this time and after eighteen hours of 
requesting participation from volunteers, the researcher had 
obtained 24 usable InfoTrac surveys and 12 usable READERS' GUIDE 
surveys. In an attempt to balance the number of InfoTrac surveys 
with READERS' GUIDE surveys, the researcher extended the time for 
surveying to another eight hours over approximately a one week 
period. 

Each month when the InfoTrac database is updated with a new 
CD-ROM, there is the possibility of records being dropped or added. 
By restricting the gathering of surveys from InfoTrac users to a 
designated two-week period, all 24 searchers searched the same 
database. The database range searched in the print indexes was 
easily framed by pointing out the volumes to be searched. Print 
indexes remain constant in records contained in each volume. Each 
volunteer searcher was asked to search for information on the 
author of a particular book, not the book itself, in order to limit 
the scope of the search. 

As seen by the extension of time for distributing READERS' 
GUIDE surveys, people were much more willing to search using 
InfoTrac. Indeed, the observation that individuals will wait for 
some time in order to use electronic databases rather than use 
print indexes created an interest in this research and was 
informally supported by the difficulty of getting volunteers. In 
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addition to this, though there was only one Inf oTrac-General 
Periodicals Index workstation available and one set of READERS' 
GUIDE, the number of searches being completed during the same time 
frame was greater for InfoTrac than for the print index. Refusals 
for volunteering to do any of the searches were high — about one out 
of four agreed to do the search. Most indicated that time was the 
crucial factor . 

The total number of surveys completed were 24 for InfoTrac 
and 20 for READERS' GUIDE. Almost all volunteers were somewhat 
reluctant, expressing lack of experience or knowledge in using 
either index • 

The first part of the survey gathered profile information from 
the volunteer searchers — characteristics which might affect the 
outcome of their search. These sought to determine 

1. if the searcher had prior experience in searching 

electronic or print indexes, 

2. if the searcher had experience with computers, and 

3 . what the age of the searcher was • 

The total sample population was 44. The sample population by age 
broke down in this way, with public school age comprising about 
23%: 

8-12 4,5% 

13-18 18.2% 

19-25 16.0% 

26-35 9.1% 

36-45 27.2% 

46+ 25.0%. 

After completing the search, the survey was completed by filling 
in responses to five questions meant to determine the searchers 
attitudes about the results. Questions #1 and #2 were open-ended. 
The searchers were asked if they felt they had found enough 
information and if they felt that they needed more help. To verify 
the responses to question #2 the searchers were asked to check a 
YES or NO response as to whether they were satisfied with the 
results of their research. Finally, a survey of the time taken for 
each search was taken. Searchers were asked to check a range of 
minutes appropriate for their search. A representation of the 
questionnaire is found in Appendix A. 



4.1 Data Analysis 

The total number of relevant records retrieved for each of 
the six searches was tabulated. In some cases, non-relevant 
records were retrieved. These were not included in computing the 
mean, median and mode scores. Table 1 illustrates these numbers 
in relationship to the number retrieved by the researcher. On 
first glance, it seems that most searchers did well as the scores 
are clustered closely together. Only the nine people searching 
for information on Kate Smith through the use of InfoTrac seem to 
have had difficulty. 
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TABLE 1 



RETRIEVAL RESULTS BY INDIVIDUAL SEARCH TOPICS 



READERS ' GUIDE TO PERIODICAL LITERATURE 



Kate Smith Robert Waller Edwards Deming 





(n=7) 


(n=8) 


(n=5) 


Mean 


2.86 


4 


9.8 


Median 


3 


4 


9 


Mode 


3 


4 


8 & 12 


Possible 


3 


6 


12 



INFOTRAC 

Kate Smith Robert Waller Edwards Deming 

(n=9) (n=8)* (n=7) 

Mean 3.11 14.66 60 

Median 1 14.75 79 

Mode 0 & 8 14 . 00 90 

Possible 8 16 90 



* two surveys not completed or used in computations 



Using answers given in part 1 of the questionnaire, the 
respondents were divided into groups of those with experience and 
those without experience in searching the respective indexes. The 
researcher set the qualification for experience in searching 
InfoTrac as having used InfoTrac more than 0-1 times. If the 
respondent indicated on the survey that he had previously used 
READERS ' GUIDE, he was considered experienced. The bar graphs in 
Figures 1 & 2 display the mean and median scores achieved by 
searchers who were perceived to have had experience and those 
without experience. 
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FIGURE 1 
Experience and Readers 1 Guide 
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The scores for both groups are similar though there is a slight 
increase in the number of records retrieved by those perceived to 
be experienced in searching those indexes • The largest difference 
in the scores within the groups shows up with the Deming search 
using InfoTrac though there is a slight difference in the Smith 
search as well. Table 1 in Appendix B displays the mean, median and 
mode scores for each group. 

Question #4 in Part 1 of the questionnaire seeks to determine 
if past computer use had impact on the success of the searcher. 
While computation of mean, median and mode scores are useful, 
precision rates compute scores by including 'wrong' answers as well 
as 'right' answers. Because the consistency of subject headings 
used in the indexes searched is important, the researcher used 
recall and precision rates for analysis of computer experience and 
success of retrieval. The formula for recall rate is: 

number of relevant articles retrieved 
number of relevant articles in the database . 

The formula for precision rate is: 

number of relevant ± non-relevant article s retrieved 
number of relevant articles in the database 

Figures 3 and 4 show computer experience in comparison with 
retrieval rates. 



FIGURE 3 
Computer Use %k InfoTrac 
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FIGURE 4 

Computer Use & InfoTrac 

number of searchers 



8 




Recall rates and precision rates are expressed as decimals. The 
closer the number is to 1.00, the better the score is. The two 
scores (recall and precision) are not dependent upon one another. 
Searcher A may have retrieved only 80% (or recall rate=0.80) of 
relevant documents but all of the documents retrieved were relevant 
so that the precision rate is 1.00. Searcher B may have retrieved 
100% of the relevant documents to obtain a recall rate of 1.00, but 
may also have retrieved several non-relevant articles which lowered 
the precision rate to something less than 1.00. 
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It would be difficult to make an assumption with population size 
and with the results shown. There seems to bo a significant 
improvement in recall and precision rates for those people wi\o use 
computers 11 or more times a month. The recall rate for those 
using computers 0-5 times a month is slightly less than for those 
using them 11 or more times a month, but the precision rate for 
articles retrieved is good. 

When examining the results of those who indicated use of 
computers 6-10 times a month , the results are less clear. Four 
our of five (80%) scored 0.13 or less on recall. This is worse 
than those with 0-5 times-a-month computer use of whom 75% 
scored 0.87 higher on recall. Three out of five (60%) of computer 
users who use them 6-10 times a month scored 0.73 on precision , 
which is poorer than the 75% of 0-5 times-a-month computer users 
who scored 0.82 or higher. 

A problem in measuring the variable of frequency of computer 
use comes in defining computer use. Computers are used in many 
ways, and a user's understanding and manipulation of computer 
processes may be low. One person may feel that use of an automated 
teller machine is computer use (which it is) but this would provide 
the lowest level of computer familiarity to the end user as 
compared with data entry, word processing and other interactive 
tasks. There is no indication on the questionnaire of the type of 
computer experience the respondents have had, though it is likely 
that those who use computers 20 or more times a month use them for 
more than repetitive tasks. 

Question #3 of the questionnaire asked for the age of the 
respondent. The purpose was to study the effects that age might 
have on success in using InfoTrac or READERS' GUIDE. Figures 5 
through 8 show the results when recall and precision rates are 
grouped according to age. These results are grouped by database 
searched. 

In InfoTrac the only group that scored consistently well was 
the 19-25 age group (n=3). For precision, 100% of this group 
scored 0.80 or higher and 100% scored 0.88 or higher on recall. 
In examination of the questionnaires, two individuals searched for 
information on Robert Waller and one searched for information on 
Kate Smith. The next best scores were achieved by the 46+ years 
age group (of whom 80% scored 0.87 or higher on recall and sixty 
percent scored 0.80 or higher on precision) and the 13-18 age group 
(of whom 60% scored 0.88 or higher on recall and 80% scored 0.82 
or higher on precision). At least four people searching InfoTrac 
found nothing while all individuals searching READERS' GUIDE found 
at least one article. In comparing figures 5 and 6 for recall 
rates and figures 7 and 8 for precision rates, it would appear that 
there is more consistency in the recall and precision rates of 
those searching READERS' GUIDE than in those searching InfoTrac. 
In using READERS' GUIDE, the recall and precision rates appear to 
increase slightly with age. Table 2 of Appendix B lists the 
subjects searched in each age group by the index searched. 
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FIGURE 5 
Recall Rate and Age 



number of te«rcher* 




o.oo .00 .« .to .so Mr .as b» .«• t.o 
Recall Rate 

i«« ©1 InioTr*o U**c 



FIGURE 6 
Recall Rate and Age 
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FIGURE 7 
Precision Rate and Age 
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FIGURE 8 

Precision Rate and Age 
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Success vs. Satisfaction 

After completing the search; the participants were asked to 
respond to questions that expressed their feelings about the 
success of their search. Question #1 asked if the searcher felt 
that he had found enough information. 

Figures 9 and 10 show that InfoTrac users were much more 
likely to feel satisfied with results than READERS GUIDE users. 
In examining the questionnaires, it seems there is a correlation 
between the total number of articles retrieved and the perception 
of finding enough information. Even though 100% of articles may 
have been found, if the number of articles retrieved is small, the 
searcher does not appear to feel he has found enough information. 



FIGURE 9 
Sufficient Information Found 
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FIGURE 10 
Sufficient Information Found 



number of ■earrher* 

6 | 




The total number of retrievable articles about KATE SMITH was 
small (three for READERS' GUIDE and eight for InfoTrac) and the 
graphs indicate that fewer people doing those searches felt enough 
information had been found* The reverse is true for the DEMING 
search where there were 12 articles in READERS' GUIDE (excluding 
articles listed under DEMING PRIZE, etc.) and at least ninety 
articles in InfoTrac. 

The Waller search seems to indicate a mid-point number for 
the perception that enough information had been found. The 
majority of READERS' GUIDE searchers were not convinced that enough 
information had been found when only six were retrievable. The 
InfoTrac searchers felt enough information had been found when 
there were a possible 16 to be retrieved. 

It's important to examine the actual recall and precision 
rates for each of the six searches before determining that the 
total number of articles retrieved is the only influencing factor 
in judging success. In Tables 2 and 3, recall and precision rates 
are higher for individuals searching InfoTrac for information on 
ROBERT WALLER than in searching READERS' GUIDE. It is surprising 
that even though individuals searching for articles on KATE SMITH 
had a very high rate of precision and recall, half of them were not 
satisfied with their results. 

In looking at the headings that people listed, some of the 
problems of retrieval could be uncovered. When using a print 
index, it seems that people were able to redirect themselves to an 
appropriate method of searching. For some reason, those using 
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TABLE 2 



READERS' GUIDE SUCCESS RATE/SATISFACTION 

by Individual Search Topic 



User Success User Satisfaction 

Recall Precision Yes No 

Smith 1.00 1.00 x 

(n=6) 1.00 1.00 x 

1.00 1.00 X 

1.00 1.00 X 

1.00 1.00 X 

0.67 1.00 X 



Waller 0.17 0.14 x 

(n=7) 0.50 0.75 X 

0.67 0.57 X 

0.67 1.00 X 

0.83 0.71 X 

0.83 1.00 no answer 

1.00 0.86 X 



Deming 0.67 1.00 x 

(n=5) 0.67 1.00 X 

0.75 1.00 X 

1.00 1.00 X 

1.00 1.00 X 
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TABLE 3 



INFOTRAC SUCCESS RATE/SATISFACTION 
by Individual Search Topics 



User Success User Satisfaction 

Recall Precision Yes No 



Smith 0.00 0.00 x 

(n=9) 0.00 0.00 x 

0.00 0.00 X 

0.13 1.00 x 

0.13 1.00 x 

0.25 0.15 X 

1.00 0.73 X 

1.00 0.80 X 

1.00 0.80 X 



Waller 0.88 0.82 x 

(n=8) 0.88 0.82 x 

0.88 1.00 x 

0.88 1.00 x 

0.92 1.00 X 

1.00 1.00 x 

1.00 0.52 X 



Deming 1.00 1.00 x 

(n=7) 1.00 1.00 x 

0.89 1.00 x 

0.88 1.00 x 

0.87 1.00 x 

0.03 1.00 X 

0.00 0.00 X 
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InfoTrac did not do that. The most common error made, that 
resulted in 0.00 recall rate, wan not inverting tho bonding. Tho 
search was entered as KATE SMITH or W. EDWARDS DEMING not SMITH, 
KATE or DEMING, W. EDWARDS. 

Question #4 asked for a yes or no response to whether the 
searcher was satisfied with the results of this search. This was 
to verify the answers to question #1. The responses were similar 
to the responses given when asked if enough information had been 
found. Results are shown in Figures 11 and 12. 



FIGURE 11 
Satisfied Users 
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FIGURE 12 
Satisfied Users 
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Judging from the reactions of the searchers they were more 
pleased with using InfoTrac than READERS' GUIDE. Table 3 of 
Appendix B breaks down the respondents feelings abut needing 
further assistance. Each index searched is broken down into age 
categories. Though most categories are equal in the perception of 
assistance needed, those people 36 years of age and older seemed 
to feel the need for help less no matter which index searched. 

Along with the number of articles retrieved, another factor 
which may have affected satisfaction with using the index was time 
needed to do the search. Figures 1 and 2 in Appendix B display the 
outcome. Even though more articles were retrieved when searching 
InfoTrac, the sc *ch time is considerably shortened thus 
reinforcing the feeling that all the necessary and accurate 
information has been retrieved. Tables 4 and 5 in Appendix B give 
the overall precision and recall rates correlated to user 
satisfaction and the index searched. Only two users of InfoTrac 
who had justifiable reasons not to be satisfied with results (0.00 
recall and 0.00 precision) expressed dissatisfaction. Sixty-five 
percent of InfoTrac searchers scored 0.87 or higher on recall. The 
rest scored 0.25 or less on recall. Six out of eight people who 
scored 0.25 or less on the recall rate expressed satisfaction with 
their results. Three of those people scored 0.15 or less on 
precision. 

Of the people who searched READERS' GUIDE, the lowest score 
was 0.17 for recall and 0.14 for precision. Fifty-five percent of 
READERS' GUIDE searchers scored higher than 0.83 in recall; 60% 
scored .75 or higher; 90% scored .66 or higher and 95% scored .50 
or higher on recall. Still, only half of the READERS' GUIDE users 
were satisfied with their results. 

The average precision rate for InfoTrac searchers was 0.73 
with an average recall rate of 0.64. For Readers' Guide, the 
average recall rate was 0.80 and average precision rate was 0.90. 
With these figures, it would seem that the perception of quantity 
and speed carry more weight in determining satisfaction than recall 
and precision. 

SEARCH ERRORS 

Close examination of the methods individuals used for their 
search and the actual headings found provides some valuable 
information for library staff who assist in reference questions. 
Of the seven people searching InfoTrac for information on W. 
Edwards Deming, only two retrieved all 90 articles. The two who 
accomplished this used the expanded search function. In their 
original search one found only one heading and the other found two. 
None of the headings were an accurate form of the name. The 
entries were DEMING, EDWARD W. , DEMING, EDWARDS W. , and DEMING, 
EDWARDS. One individual retrieved 80 article by locating two 
headings: the main heading DEMING. W. EDWARDS and a variant located 
immediately before DEMING, W. EDWARD. Two others retrieved 78 
articles by finding the main heading. One of those found an 
additional article by searching the book title, OUT OF THE CRISIS. 
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This article would have been retrieved had the individual located 
the heading DEMMING, EDWARDS. One individual first searched under 
W. EDWARDS DEMING; then under DEMING, EDWARDS and wrote down 
"wrong guy" to end up with no retrievals. 

The concern with the headings for W. Edwards Deming was that 
there were too many variants and that information would be lost. 
The concern with the headings for Robert Waller and Kate Smith were 
that the headings were indiscriminate and would lead to retrieving 
inaccurate information. 

Eight people searched for information on Robert Waller. Of 
those, only two retrieved information under the two headings with 
relevant information. The first individual searched under WALLER, 
R.J. then located the heading WALLER, ROBERT J. and noticed that 
it was not the correct one. This individual did correctly retrieve 
the relevant articles under WALLER, ROBERT and WALLER, ROBERT 
JAMES. The second individual located the two headings and 
retrieved the relevant articles but also retrieved 15 non-relevant 
articles which brought the precision rate down considerably. Three 
of the eight searchers located only one heading (WALLER, ROBERT 
JAMES) so that two of the 16 articles were missed. Reducing 
precision, the last two searchers retrieved one non-relevant 
article each and 14 of the relevant articles. 

Finally, the greatest number of errors were made by the group 
searching for information on Kate Smith. As stated before, there 
was only one heading for SMITH, KATE but there were two different 
individuals listed under the one heading. Three of the nine 
individuals found.no information. The poor showing was a result of 
entering the search incorrectly as KATE SMITH, not inverted as it 
should be. As a result, one of these individuals retrieved two 
articles under the heading KATE ON THE COAST, which is a book. In 
all, six of the searchers first entered the name as KATE SMITH; two 
of these then inverted the entry to SMITH, KATE. Not one of the 
searchers had a completely successful search in that even the 
searchers who retrieved the eight relevant articles retrieved non- 
relevant articles. Two retrieved the other two out of ten articles 
listed under SMITH, KATE and one retrieved a total of three non- 
relevant articles. The oldest article under the heading SMITH, 
KATE, and that was concerning the singer Kate Smith, was her 
obituary notice. Two searchers retrieved only this article. The 
last person first searched KATE SMITH, then SMITH, KATE and SMITH, 
K. This led to the retrieval of two relevant articles and five 
non-relevant articles. 

CONCLUSION 

It is still apparent that there is a great deal of 
satisfaction on the part of the general public about using 
InfoTrac, while the feelings are mixed about using READERS' GUIDE. 
The likelihood that electronic databases and indexes would loose 
their appeal seems unlikely. The perception that they are 
completely user-friendly and can stand alone as a reference 
resource should be considered an inaccurate assumption. As this 
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small study indicates there is the danger of sending individuals 
out of the library with misinformation. Looking again at Tables 
2 and 3, it can be seen that there are some differences in quality 
of retrieval for the unassisted searcher • if we are to face the 
reality of the future with more self -assist work stations, we must 
also assume some responsibility in providing accurate information 
that will not mislead. When 30% of the searchers using InfoTrac 
retrieve articles that are not relevant to the subject searched, 
and the average recall rate is 0.64, perhaps we should forewarn 
those individuals waiting in line to use the index. The reference 
interview should not be considered complete by waving the 
questioner in the direction of the nearest InfoTrac workstation 
because it is so quick to use, because we are understaffed and 
because we, as more experienced searchers, have forgotten what 
others do not know. Another aspect of this topic is 'How 
appropriate is this database for this search?' • In comparing the 
periodicals indexed in each of these databases, it appears we may 
be letting our patrons down by not guiding them in the direction 
of the best sources . Would we send someone to the MCGRAW-HILL 
ENCYCLOPEDIA OF SCIENCE AND TECHNOLOGY for information on religions 
of the West Indies? The comparison is simplified but the 
implications are as great. 
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InfoTrac/Readers' Guide to Periodical Literature 

User Questionnaire 

This survey will be used to analyze the use of two general 
periodical indexes available in this library. Your assistance in 
submitting accurate responses will be appreciated. There will be 
no further obligation on your part. 

1. How many other times have you used InfoTrac on CD-ROM before? 

0-1 times 

2-5 times 

6-10 times 

More than 10 times_ 

2. Please check other indexes that you have used. 

Readers' Guide to Periodical Literature (print) 

Cleveland News Index (Plain Dealer) (print) 

New York Times Index (print) 

Wall St. Journal Index (print) 

Compact Disclosure (CD-ROM) _ - 

Wilsondisc (CD-ROM) 

InfoTrac (CD-ROM) 

InfoTrac (online) - — 

Other - 



3. What is your age? 

8-12 

13-18 

19-25 

26-35 

36-45 

46 or older 

4. How often do you use computers? 

never 

0-5 times a months 

6-10 times a month. 

11-20 times a month 

21 or more times a month 

InfoTrac and Readers 9 Guide to Periodical Literature are indexes 
that help people find articles in magazines and newspapers. They 
are arranged alphabetically by subject. You are being asked to 
find information on a specific person in one of those indexes. 
Please follow the instructions on the following page. 
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INFOTRAC USERS: 

1. The next page in this packet is a photocopy of the title page 
of a book. Please find all periodical (magazine and newspaper) 
articles that are listed in InfoTrac about this AUTHOR . 

2. Please write down each name/subject heading that you look up 
even if it is very similar to other searches that you tried and 
even if you do not find the name/subject listed. 

If there is no listing for that name/subject, write a 0 (zero) 
next to it. 

3. When you find a name/subject heading that you think will give 
you the information you need, make a print-out of every article 
listed under it that you feel applies to the author. 

You may make a print-out by pressing the letter 'p' on the 
keyboard or the key 'F3' on the keyboard. 

4. When you have finished, attach all of the print-outs to the 
questionnaire and return it to the reference desk. 

Please list name/subjects searched here. 

SUBJECT EXPANDED SEARCH (optional) 



1. Do you feel that you found enough information? 



2. Do you feel that you needed more help? 



3. Did you receive any help today in doing this search? 
no 

yes from: friend family staff other 



4. Were you satisfied with the results? no yes 



5. How long did the search take? 0-5 minutes. 

6-10 minutes. 
11-15 minutes. 
16-20 minutes 
over 20 minutes. 
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READERS ' GUIDE TO PERIODICAL L ITERAT URE USERS: 

1. In the back of this packet is a photocopy of the title page of 
a book. Please find all periodical (magazine and newspaper) 
articles that are listed in READERS 'GUIDE about this AUTHOR. 

2. Please write down each name/subject heading that you look up 
even if it is very similar to other searches that you have tried 
and even if you do not find the name/subject listed. 

If there is no listing for that name/subject, write a 0 (zero) 
next to it. 

3. Begin with volume 46 for the year 1986. Write down each 
name/subject you try. When you find a name/subject heading that 
lists articles you feel are about this AUTHOR . write down: 

(1) the last name of the person who wrote the article 

(2) the name of the periodical that the article appears in 

(3) the date of the periodical 

(4) the page number in the READERS' GUIDE volume you are 
using. 

4. After you have finished looking through this volume, go on to 
the next volume - Number 47. 

5. Continue this process until you have searched through all 
volumes from 1986- to the most current volume. 

Volume 46 

Subject Author Magazine Date Page # 



Volume 47 

Subject Author Magazine Date Page # 



Volume 48 

Subject Author Magazine Date Page # 
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Volume 49 (1989) 

Subject Author Magazine Dato Pngo # 



Volume 50 (1990) 

Subject Author Magazine Date Page # 



Volume 51 (1991) 

Subject Author Magazine Date Page # 



Volume 52 (1992) 

Subject Author Magazine Date Page # 



Volume 93 (1993) 

Subject Author Magazine Date Page # 



1. Do you feel that you found enough information?. 
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2, Do you feel that you needed more help? 



3, Did you receive any help today in doing this search? 
no 

yes from: friend family staff other 



4, Were you satisfied with the results? no yes. 



5, How long did the search take? 

0-5 minutes_ 
6-10 minutes^ 
11-15 minutes_ 
16-20 minutes_ 
over 20 minutes_ 
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(Appendix U) 



TABLE 1 



PRIOR EXPERIENCE AND RETRIEVAL RESULTS 



READERS ' GUIDE 



No Experience 



Experienced 



Smith 


Waller 


Deming 


Smith 


Waller 


Deming 


(n=5) 


(n=3) 


(n=l) 


(n=2) 


(n=5) 


(11=4) 


mean 2 . 8 


3 .66 


8 


3 


4.2 


10.25 


median 3 


4 


8 


3 


5 


10.5 


mode 3 


4 


8 


3 


5 


12 


possible 3 


6 


12 


3 


6 


12 






INFOTRAC 








No Experience 




Experienced 




Smith 


Waller 


Deming 


Smith 


Waller 


Deming 


(11=5) 


(n=4)* 


(11=3) 


(11=4) 


( n=4 ) * 


(11=4) 


mean 2 . 4 


14 


27 


4 


15.3 


84.75 


median 1 


14 


3 


4 


16 


85 


mode. 1 


14 


0,3,78 


0 & 


8 16 


90 


possible 8 


16 


90 


8 


16 


90 


* one survey not complete or used in survey 
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TABLE 2 



SUBJECT/NAMES SEARCHED BY AGE AND DATABASE 



Readers 7 Guide 



O X c* • • • • 


1 


Smith 


1 


Waller 






13-18 .... 


1 


Smith 


2 


Waller 






1 Q-OR .... 

-L- -S 6 J • • • • 


1 


Smith 


2 


Waller 


1 


Deming 


6 v ^ • • • • 


1 


Smith 










36-4^ . . . . 

J U m ~t •••• 


1 


Smith 






4 


Deming 


46+ 


2 


Smith 


3 


Waller 






InfoTrac 














8-12 


0 












13-18 


2 


Smith 


1 


Waller 


2 


Deming 


19-25 


1 


Smith 


2 


Waller 






26-35 


1 


Smith 


1 


Waller 


1 


Deming 


36-45 


2 


Smith 


3 


Waller 


2 


Deming 


46+ 


3 


Smith 


1 


Waller 


2 


Deming 
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TABLE 3 





ASSISTANCE 


NEEDED 


BY AGE GROUP 


Reader's Guide 








8-12 


YES 
1 


NO 

1 


MAYBE 


13-18 


1 


2 




19-25 


3 


1 




26-35 


1 






36-45 


1 


4 




46+ 


1 


4 




InfoTrac 








8-12 


YEo 


MA 

Nvl 


MAYRF. 


13-18 


2 


3 




19-25 




3 




26-35 


2 


1 




36-45 


1 


6 




46+ 




5 


i 



0 

ERJC 



46 



TABLE 4 



INFOTRAC SUCCESS RATE/SATISFACTION 
(n=24)* 



User Success 
Recall Precision 



User Satisfaction 
Yes No 



1.00 


1.00 


X 


1.00 


1.00 


X 


1.00 


1.00 


X 


1.00 


0.80 


X 


1.00 


0.80 


X 


1.00 


0.73 


X 


1.00 


0.56 


X 


0.92 


1.00 




0.89 


1 .00 


X 


0.88 


1.00 


X 


0.88 


1 .00 


X 


0.88 


1 .00 


X 


0.88 


0.82 


X 


0.88 


0.82 


X 


0.87 


1.00 


X 


0.25 


0.15 


X 


0.13 


1.00 


X 


0.13 


1.00 


X 


0.03 


1 .00 


X 


0.00 


0.00 




0.00 


0.00 




0.00 


0.00 


X 


0.00 


0.00 


X 



X 
X 



* one survey not completed or used in survey 
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TABLE 5 
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READERS' GUIDE SUCCESS RATE/SATISFACTION 

(n=20) 

User Success User Satisfaction 



Recall 


Precision 


Yes 


No 


0.17 


0.14 




X 


0.50 


0.75 


X 




0.66 


1.00 




X 


0.67 


0.57 




X 


0.67 


1.00 




X 


0.67 


1.00 




X 


0.67 


1.00 


X 




0.67 


1.00 


X 




0.75 


1.00 


X 




0.83 


0.71 




X 


0.83 


1.00 


no answer 




1.00 


0.86 


X 




1.00 


1.00 


X 




1.00 


1.00 


X 




1.00 


1.00 


X 




1.00 


1.00 


X 




1.00 


1.00 


X 




1.00 


1.00 




X 


1.00 


1.00 




X 


1.00 


1.00 




X 
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FIGURE 1 
Search Time/Readers' Guide 
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FIGURE 3 
Readers' Guide Help Needed 
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(Appendix C) 



READERS GUIDE TO PERIODICAL LITERATURE 



SMITH, KATE 

Vol 46 (1986) 
obit 



Newsweek 6/30/86 



1740 



Vol 47 (1987) 



Peop Wk 
New York 



6/15/87 
10/19/87 



1749 
1749 



Vol 
Vol 
Vol 
Vol 
Vol 
Vol 



48 
49 
50 
51 
52 
53 



(1988) 
( 1989) 
(1990) 
(1991) 
(1992) 
(1993) 



0 
0 
0 
0 
0 
0 



WALLER, . 
Vol 
Vol 
Vol 
Vol 



46 
47 
48 
49 



(1986) 
( 1987) 
(1988) 
(1989) 



0 
0 
0 
0 



Vol 50 (1990) 

WALLER, ROBERT, 



1913- (not applicable) 
Hist Today 1/90 



1975 



Vol 51 (1991) 

WALLER, ROBERT 



Read Dig 



2/91 



2075 



Vol 52 (1992) 

WALLER, ROBERT J. 

Mutter 



Newsweek 9/7/92 2181 
Pub Wkly 7/20-21/92 2181 
Peo Wkly 11/16/92 2181 



Vol 53 (1993) Jan-April 

WALLER, ROBERT JAMES, 1939- 
Tilsner Busi Wk 



3/1/93 



744 



Vol 53 (1993) May-June 

WALLER, ROBERT JAMES, 1939- 

Teavis Succ Farm May/June 



747 
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DEMING , DEMMING 

Vol 46 (1986) - 0 

Vol 47 (1987) 

DEMING, W. EDWARDS 
Jones 

Vol 48 (1988) 

DEMING PRIZE 
Gabor 
Impoco 

Vol 49 (1989) 

DEMING, W. EDWARDS 
Moreau 

Vol 50 (1990) 



Scho Upd 4/6/87 



U.S.News 11/28/88 
U.S.News 11/28/88 



Cha Times Sept 1989 



594 



570 
570 



573 



— — f - — 

Dobyns 


Smithson 


Aug 1990 




567 


DEMING PRIZE 










Dusky 


Work Worn 


July 1990 




c r *7 
S6 / 


Bacon 


Nat Busi 


Jan 1990 




567 


51 (1991) 










DEMING, W. EDWARDS 










Kraar 


Fortune 


Spr/Sum 1991 


594 


Austin 


Work Worn 


9/1/91 




594 


Port 


Bus Week 


10/25/91 




594 


Parshall 


U.S. News 


4/22/91 




594 


Linden 


Forbes 


5/27/91 




594 


DEMING PRIZE 








594 


Woods 


Forbes 


3/18/91 




Nef f 


Bus Wk 


10/25/91 




594 


Fla. Power 


Fortune 


7/1/91 




594 


52 (1992) 
0 










93 (1993) Jan-April 










DEMING, W. EDWARDS 










Blankstein 


Educ Dig 


Feb 93 


218 




Tetzell 


Fortune 


1/11/93 


218 




Holt 


PhiDeKap 


Jan 93 


218 




Schmoker ,Wilso 


PhiDeKap 


Jan 93 


218 





Vol 53 (1993) April-June 

DEMING, W. EDWARDS - 0 
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